{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0 install numpy, matplotlib, pillow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 安装numpy数值计算包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install matplotlib\n",
    "!pip install pillow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "安装matplotlib/pyplot显示图像"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Activation Function（激活函数）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### ReLU函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ReLU函数是Softplus函数的钝化版本。ReLU函数有几种推广版本：\n",
    "\n",
    "绝对值整流（absolute value rectification）。\n",
    "\n",
    "渗漏整流线性单元（Leaky ReLU）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ReLU函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ReLU函数是Softplus函数的钝化版本。ReLU函数有几种推广版本：\n",
    "\n",
    "绝对值整流（absolute value rectification）。\n",
    "\n",
    "渗漏整流线性单元（Leaky ReLU）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(-5,5,100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "relu = lambda x: np.maximum(x, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(x, relu(x), color='blue', lw=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### PReLU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "$ReLU(x) = max(0,x)$\n",
    "\n",
    "参数化整流线性单元（parametric ReLU）或者PReLU。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "x = np.linspace(-5,5,100)\n",
    "pr = lambda xs: [x*0.1 if x <0 else x for x in xs] \n",
    "\n",
    "plt.plot(x, pr(x), color='blue', lw=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Softplus函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Softplus函数是ReLu函数的软化版本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(-5,5,100)\n",
    "softplus = lambda x: np.log(1.0 + np.exp(x))\n",
    "plt.plot(x, softplus(x), color='blue', lw=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### sigmoid函数\n",
    "\n",
    "$sigmoid(x)=\\frac{1}{1+e^{-x}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(-5,5,100)\n",
    "sigmoid = lambda x: 1 / (1 + np.exp(-x))\n",
    "plt.plot(x,sigmoid(x), color='red', lw=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 饱和型S函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(-5,5,100)\n",
    "pr = lambda xs: [0 if x <0 else 1 for x in xs] \n",
    "plt.plot(x, pr(x), color='blue', lw=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### logit函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(-5,5,100)\n",
    "logit = lambda x: np.log(x / (1 - x))\n",
    "plt.plot(x,logit(x), color='blue', lw=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SoftSign函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(-5,5,100)\n",
    "softsign = lambda x: x / (1 + np.absolute(x))\n",
    "plt.plot(x,softsign(x), color='blue', lw=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tanh函数\n",
    "\n",
    "$tanh(x)=\\frac{e^x-e^{-x}}{e^x+e^{-x}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(-5,5,100)\n",
    "th = lambda x: np.tanh(x)\n",
    "plt.plot(x,th(x), color='blue', lw=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### hard tanh函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hard_tanh(x):\n",
    "    if abs(x)<1:\n",
    "        return x\n",
    "    else:\n",
    "        if x>1: \n",
    "            return 1\n",
    "        else:\n",
    "            return -1\n",
    "x = np.linspace(-5,5,100)\n",
    "htan = lambda xs: [hard_tanh(x) for x in xs] \n",
    "plt.plot(x,htan(x), color='blue', lw=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### softmax函数\n",
    "$g(z_m)=\\frac{e^{z_m}}{\\sum_k {e^{z_k}}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "w=[1,2,3,4,5,6,7,8,9]\n",
    "w_exp=[math.exp(i) for i in w]\n",
    "print(w_exp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sum_w_exp = sum(w_exp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "softmax = [round(i / sum_w_exp, 3) for i in w_exp]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(softmax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sum(softmax))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(softmax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
